Rasch in R and Quarto, part 2

PSS-10

Author
Affiliation
Magnus Johansson
Published

June 13, 2022

1 Background information

The Perceived Stress Scale (Lee 2012; Nordin and Nordin 2013) is a measure of subjective stress. It is available in three versions, with 14, 10 or 4 items. This dataset consists of the 10 item version, PSS-10.

All items have the same response options, Aldrig, Nästan aldrig, Ibland, Ganska ofta, Väldigt ofta, which are recoded to 0-4 for this analysis. The questions are introduced with the text Under den senaste månaden, hur ofta har du:.

The table below lists all items, with the four positively worded items highlighted with green backgrounds.

itemnr item
q1 varit upprörd över något som hände helt oväntat?
q2 känt att du var oförmögen att kontrollera de viktiga sakerna i ditt liv?
q3 känt dig nervös och “stressad”?
q4 litat på din förmåga att hantera dina personliga problem?
q5 känt att saker och ting har gått din väg?
q6 känt att du inte kunnat hantera allt som du måste göra?
q7 klarat av att kontrollera irritationsmoment i ditt liv?
q8 känt att du haft kontroll?
q9 varit arg över sådant som hänt och varit utanför din kontroll?
q10 känt att svårigheter hopat sig så att du inte kunnat hantera dem?

1.1 Participants

We have 525 participants and their gender distribution is shown in the table below. All participants are Swedish adults and the dataset was collected to validate a measure of social interactions at workplaces (Johansson and Biglan 2021; Johansson 2021).

Gender n Percent
Female 129 24.6
Male 396 75.4
Age distribution
Age group n Procent
18-29 51 9.7
30-39 162 30.9
40-49 133 25.3
50-59 143 27.2
60+ 36 6.9

Gender and age distribution in sample

Demographic subgroups are too small for properly powered DIF-analyses, which should have at least 150 in each subgroup, and ideally 250 or more.

2 Descriptives of raw data

Responses to all items are summarized below.

Response category Number of responses Percent
0 984 18.7
1 2359 44.9
2 1402 26.7
3 417 7.9
4 88 1.7

3 Descriptives - item level

3.0.1 Recoding responses

Since there are fewer than 10 responses in category 4 for most items we will merge category 4 with category 3 before proceeding with the analysis.

4 Analysis of response categories

4.1 ICC plots for recoded items

Response categories are working as expected, after the adjustments made.

5 Dimensionality

The eRm package, which uses Conditional Maximum Likelihood (CML) estimation, will be used primarily. For this analysis, the Partial Credit Model will be used.

First, we look at the eigenvalues from PCA analysis of Rasch residuals. These should be below 2.0 to support unidimensionality. If the first eigenvalue is above 2.0, the subsequent analysis will include a clustering analysis, using Mokken Scaling.

PCA of Rasch model residuals
Eigenvalues
2.20
1.50
1.13
1.05
1.01

The first eigenvalue exceeds 2.0, which indicates multidimensionality.

OutfitMSQ InfitMSQ OutfitZSTD InfitZSTD
q1 0.927 0.917 -0.923 -1.057
q2 0.763 0.757 -3.28 -3.527
q3 0.868 0.865 -1.87 -1.914
q4 1.146 1.144 1.693 1.829
q5 1.02 0.996 0.39 0.116
q6 1.023 0.999 0.435 0.156
q7 1.115 1.084 1.19 0.907
q8 0.813 0.793 -2.092 -2.355
q9 0.935 0.908 -1.195 -1.416
q10 0.692 0.698 -4.36 -4.341

Values in red are beyond the pre-set cutoff values. We’ll leave them for now.

5.1 Residual correlations

A correlation matrix is created based on the Rasch model residuals for each item.

The average correlation is calculated, and item pairs that correlate more than the pre-set cutoff value of 0.2 above the average correlation are indicated in red in the table below.

q1 q2 q3 q4 q5 q6 q7 q8 q9 q10
q1
q2 -0.01
q3 -0.05 -0.04
q4 -0.24 -0.21 -0.17
q5 -0.2 -0.2 -0.18 0.17
q6 -0.06 -0.02 0.03 -0.3 -0.23
q7 -0.26 -0.17 -0.27 0.22 0.1 -0.27
q8 -0.34 -0.09 -0.05 0.07 0.1 -0.06 0.11
q9 0.24 -0.07 -0.19 -0.25 -0.14 -0.14 -0.2 -0.25
q10 -0.1 0.06 0.04 -0.16 -0.23 0 -0.19 -0.03 0.06
Note:
Relative cut-off value (highlighted in red) is 0.107, which is 0.2 above the average correlation.

We have multiple item pairs with too large correlations:

  • q4 with 5 and 7
  • q7 with 8
  • q1 with 9

This means that all four positively worded items are problematic. They will be removed, and a new analysis of item fit and residual correlations conducted.

5.2 Item fit 2, without positive items

PCA of Rasch model residuals
Eigenvalues
1.63
1.25
1.17
1.12
0.82

Eigenvalues look good, supporting unidimensionality.

OutfitMSQ InfitMSQ OutfitZSTD InfitZSTD
q1 0.872 0.854 -2.216 -2.563
q2 0.805 0.789 -3.404 -3.786
q3 0.921 0.9 -1.316 -1.736
q6 1.005 0.983 0.107 -0.277
q9 0.905 0.902 -1.626 -1.679
q10 0.707 0.724 -5.232 -5.015
itemnr item
q1 varit upprörd över något som hände helt oväntat?
q2 känt att du var oförmögen att kontrollera de viktiga sakerna i ditt liv?
q3 känt dig nervös och “stressad”?
q6 känt att du inte kunnat hantera allt som du måste göra?
q9 varit arg över sådant som hänt och varit utanför din kontroll?
q10 känt att svårigheter hopat sig så att du inte kunnat hantera dem?

Item fit is acceptable for all items.

5.3 Residual correlations 2

q1 q2 q3 q6 q9 q10
q1
q2 -0.18
q3 -0.23 -0.16
q6 -0.26 -0.16 -0.11
q9 0.1 -0.22 -0.35 -0.32
q10 -0.29 -0.05 -0.08 -0.14 -0.07
Note:
Relative cut-off value (highlighted in red) is 0.032, which is 0.2 above the average correlation.

Item 1 and 9 are still correlated above the cutoff. Both items contain wording about being upset/angry about things that are unexpected or outside of one’s control.

Both items have similar item fit. Let’s look at the targeting properties of these items to see how they compare.

5.4 Targeting

Items 1 and 9 are virtually identical in threshold locations, let’s see how the DIF analysis works out before removing either of them.

6 DIF analysis of gender

[1] "No significant DIF found."

No significant gender DIF is indicated.

6.1 DIF analysis of age

We will look at the age variable, for practice.

Item 2 3 Mean location StDev MaxDiff
q1 0.210 0.000 0.140 0.121 0.210
q2 0.445 0.335 0.297 0.171 0.110
q3 -1.231 -0.332 -0.222 1.069 0.898
q6 -0.760 -0.976 -0.507 0.636 0.217
q9 0.501 -0.076 0.334 0.357 0.577
q10 0.834 1.050 0.700 0.433 0.216

Finally we get to investigate DIF size! One interesting property of the Rasch Tree DIF method is that when a DIF variable has more than 2 levels, it will determine where the largest differences are, and build a sequential tree if multiple significant differences are found.

In our sample, there was DIF of age found between those with category 2 or lower (age 18-29), and those with category 3 and above (age 30+).

The DIF table highlights values beyond 0.5 logits maximum difference between groups. If more than two groups are included in the DIF analysis, the lowest and highest value is automatically identified.

6.1.1 DIF size figure

Significant DIF size was found for items 3 and 9. This could be a good reason to remove item 9 instead of item 1 (residual correlation problems earlier). Our sample is too small to draw conclusions about DIF, but since one of them needs to be removed, item 9 will be removed before moving on to the Targeting section.

6.2 DIF interaction (age + gender)

There is another unique function in the Rasch Tree DIF, which is the interaction of multiple DIF variables. We can input age and gender at the same time! Our groups will be way too small, so this is really only to illustrate how to do it.

Code
# set up the DIF analysis
df.tree <- data.frame(matrix(ncol = 0, nrow = nrow(df.omit.na))) # we need to make a new dataframe
df.tree$difdata <- as.matrix(df.omit.na) # containing item data in a nested dataframe
# and DIF variables:
df.tree$age<-dif.age
df.tree$gender<-dif.gender
pctree.out<-pctree(difdata ~ age + gender, data = df.tree)
plot(pctree.out)

No interaction DIF found (only age shows problems), which unfortunately means we won’t get to make a neat table showing multiple groups, item locations, means and standard deviations.

7 Targeting

7.1 Reliability

“Test/scale information” shows the collective information based on the combination of all items into a unidimensional scale.

Reliability is insufficient for the 5 item PSS scale.

7.1.1 Item information

“Item information” shows individual curves indicating the amount of information contributed by each item.

7.2 Person fit

8 Item parameters

Item thresholds and locations
Threshold 1 Threshold 2 Threshold 3 Item location
q1 -1.45 0.79 2.83 0.73
q2 -1.02 1.20 2.97 1.05
q3 -1.69 0.33 2.21 0.28
q6 -2.55 0.08 1.70 -0.26
q10 -0.70 1.86 3.98 1.71

9 Ordinal to interval transformation table

Since the scale is not reliable enough, this will not be provided.

10 Software used

Package Version Citation
base 4.1.2 @base
car 3.0.12 @car
cowplot 1.1.1 @cowplot
eRm 1.0.2 @eRm2007b; @eRm2007c; @eRm2009d; @eRm2013e; @eRm2015f; @eRm2019g; @eRm2021a
formattable 0.2.1 @formattable
grateful 0.1.11 @grateful
HH 3.1.47 @HH2004; @HH2014; @HH2015; @HH2022
kableExtra 1.3.4 @kableExtra
knitr 1.39 @knitr2014; @knitr2015; @knitr2022
matrixStats 0.61.0 @matrixStats
mirt 1.36.1 @mirt
mokken 3.0.6 @mokken2007; @mokken2012
psych 2.2.3 @psych
psychotree 0.15.4 @psychotree2010e; @psychotree2011a; @psychotree2015b; @psychotree2018c; @psychotree2018d
reshape 0.8.8 @reshape
RISEkbmRasch 0.1.2 @RISEkbmRasch
rmarkdown 2.14 @rmarkdown2018; @rmarkdown2020; @rmarkdown2022
tidyverse 1.3.1 @tidyverse

11 References

Johansson, Magnus. 2021. “Dataset for the Initial Development of the Group Nurturance Inventory.” figshare. https://doi.org/10.6084/M9.FIGSHARE.13042010.V3.
Johansson, Magnus, and Anthony Biglan. 2021. “The Group Nurturance Inventory Initial Psychometric Evaluation Using Rasch and Factor Analysis.” BMC Public Health 21 (1): 1454. https://doi.org/10.1186/s12889-021-11474-5.
Lee, Eun-Hyun. 2012. “Review of the Psychometric Evidence of the Perceived Stress Scale.” Asian Nursing Research 6 (4): 121–27. https://doi.org/10.1016/j.anr.2012.08.004.
Nordin, Maria, and Steven Nordin. 2013. “Psychometric Evaluation and Normative Data of the Swedish Version of the 10-Item Perceived Stress Scale.” Scandinavian Journal of Psychology 54 (6): 502–7. https://doi.org/10.1111/sjop.12071.